Grounding of Textual Phrases in Images by Reconstruction

Grounding (i.e. localizing) arbitrary, free-form textual phrases in visual content is a challenging problem with many applications for human-computer interaction and image-text reference resolution. Although many data sources contain images which are described with sentences or phrases, they typically do not provide the spatial localization of the phrases. This is true for both curated datasets such as MSCOCO [23] or large user generated content as e.g. in the YFCC 100M dataset [31]. Consequently, being able to learn from this data without grounding supervision would allow large amount and variety of training data. For this setting we propose GroundeR, a novel approach, which is able to learn the grounding by aiming to reconstruct a given phrase using an attention mechanism. More specifically, during training time, the model encodes the phrase using an LSTM, and then has to learn to attend to the relevant image region in order to reconstruct the input phrase. At test time the correct attention, i.e. the grounding is evaluated. On the Flickr 30k Entities dataset [26] our approach outperforms prior work which, in contrast to us, trains with the grounding (bounding box) annotations.